We introduce the Conditional Independence Regression CovariancE (CIRCE), a measure of conditional independence for multivariate continuous-valued variables. CIRCE applies as a regularizer in settings where we wish to learn neural features $\varphi(X)$ of data $X$ to estimate a target $Y$, while being conditionally independent of a distractor $Z$ given $Y$. Both $Z$ and $Y$ are assumed to be continuous-valued but relatively low dimensional, whereas $X$ and its features may be complex and high dimensional. Relevant settings include domain-invariant learning, fairness, and causal learning. The procedure requires just a single ridge regression from $Y$ to kernelized features of $Z$, which can be done in advance. It is then only necessary to enforce independence of $\varphi(X)$ from residuals of this regression, which is possible with attractive estimation properties and consistency guarantees. By contrast, earlier measures of conditional feature dependence require multiple regressions for each step of feature learning, resulting in more severe bias and variance, and greater computational cost. When sufficiently rich features are used, we establish that CIRCE is zero if and only if $\varphi(X) \perp \!\!\! \perp Z \mid Y$. In experiments, we show superior performance to previous methods on challenging benchmarks, including learning conditionally invariant image features.

当部署在与受过训练的域不同的域中时，机器学习方法可能是不可靠的。为了解决这个问题，我们可能希望学习以域不变性的数据表示，即我们保留跨域稳定但抛出虚假变化的部分的数据结构。这种类型有许多表示学习方法，包括基于数据增强，分配不变性和风险不变性的方法。不幸的是，当面对任何特定的现实世界转移时，目前尚不清楚这些方法中有哪些（如果有的话）可以正常工作。本文的目的是展示不同方法如何相互关系，并阐明各自预期成功的现实情况。关键工具是一个新的域转移概念，它依靠因果关系是不变的想法，但是非因果关系（例如，由于混杂而引起的）可能会有所不同。

现实世界的分类问题必须与域移位竞争，该域移动是部署模型的域之间的（潜在）不匹配以及收集训练数据的域。处理此类问题的方法必须指定域之间哪种结构与什么变化。一个自然的假设是，因果关系（结构）关系在所有领域都是不变的。然后，很容易学习仅取决于其因果父母的标签$ y $的预测指标。但是，许多现实世界中的问题是“反农产品”，因为$ y $是协变量$ x $的原因 - 在这种情况下，$ y $没有因果父母，而天真的因果不变性是没有用的。在本文中，我们研究了在特定的域转移概念下的表示形式学习，该概念既尊重因果不变性又自然处理“反毒物”结构。我们展示了如何利用域的共享因果结构来学习一个表示不变预测因子的表示，并且还允许在新域中快速适应。关键是将因果假设转化为学习原理，这些学习原理“不变”和“不稳定”特征。关于合成数据和现实世界数据的实验证明了所提出的学习算法的有效性。代码可在https://github.com/ybjiaang/actir上找到。

科学研究的基本目标是了解因果关系。然而，尽管因果关系在生活和社会科学中的重要作用，但在自然语言处理（NLP）中并不具有相同的重要性，而自然语言处理（NLP）传统上更加重视预测任务。这种区别开始逐渐消失，随着因果推理和语言处理的融合，跨学科研究的新兴领域。尽管如此，关于NLP因果关系的研究仍然散布在没有统一的定义，基准数据集的情况下，并清楚地表达了将因果推论应用于文本领域的挑战和机遇，并具有其独特的属性。在这项调查中，我们巩固了整个学术领域的研究，并将其置于更广泛的NLP景观中。我们介绍了用文本估算因果效应的统计挑战，其中包含文本用作结果，治疗或解决混杂问题的设置。此外，我们探讨了因果推理的潜在用途，以提高NLP模型的鲁棒性，公平性和解释性。因此，我们提供了NLP社区因果推断的统一概述。

非正式地，“虚假关联”是模型对分析师认为无关紧要的输入数据的某些方面的依赖性。在机器学习中，这些都有一个知道它 - 当你看到它的字符;例如，改变句子主题的性别改变了情绪预测因素的输出。要检查虚假相关性，我们可以通过对输入数据的无关部分进行扰动并查看模型预测变化来“压力测试”模型。在本文中，我们使用因果推断的工具研究压力测试。我们将反事实不变性介绍，作为一个正式化的要求，即改变输入不相关的部分不应改变模型预测。我们将反事实不变性与域外模型性能进行连接，并提供用于学习（大约）反事实不变预测器的实用方案（无需访问反事实示例）。事实证明，反事实不变性的手段和含义都基本上取决于数据的真实潜在的因果结构 - 特别是标签是否导致特征或功能导致标签。不同的因果结构需要不同的正则化方案，以诱导反事实不变性。同样，反事实不变性暗示不同的域移位保证，具体取决于底层的因果结构。该理论是通过文本分类的经验结果支持。

Selecting the number of topics in LDA models is considered to be a difficult task, for which alternative approaches have been proposed. The performance of the recently developed singular Bayesian information criterion (sBIC) is evaluated and compared to the performance of alternative model selection criteria. The sBIC is a generalization of the standard BIC that can be implemented to singular statistical models. The comparison is based on Monte Carlo simulations and carried out for several alternative settings, varying with respect to the number of topics, the number of documents and the size of documents in the corpora. Performance is measured using different criteria which take into account the correct number of topics, but also whether the relevant topics from the DGPs are identified. Practical recommendations for LDA model selection in applications are derived.

Applying deep learning concepts from image detection and graph theory has greatly advanced protein-ligand binding affinity prediction, a challenge with enormous ramifications for both drug discovery and protein engineering. We build upon these advances by designing a novel deep learning architecture consisting of a 3-dimensional convolutional neural network utilizing channel-wise attention and two graph convolutional networks utilizing attention-based aggregation of node features. HAC-Net (Hybrid Attention-Based Convolutional Neural Network) obtains state-of-the-art results on the PDBbind v.2016 core set, the most widely recognized benchmark in the field. We extensively assess the generalizability of our model using multiple train-test splits, each of which maximizes differences between either protein structures, protein sequences, or ligand extended-connectivity fingerprints. Furthermore, we perform 10-fold cross-validation with a similarity cutoff between SMILES strings of ligands in the training and test sets, and also evaluate the performance of HAC-Net on lower-quality data. We envision that this model can be extended to a broad range of supervised learning problems related to structure-based biomolecular property prediction. All of our software is available as open source at https://github.com/gregory-kyro/HAC-Net/.

Counterfactual explanation is a common class of methods to make local explanations of machine learning decisions. For a given instance, these methods aim to find the smallest modification of feature values that changes the predicted decision made by a machine learning model. One of the challenges of counterfactual explanation is the efficient generation of realistic counterfactuals. To address this challenge, we propose VCNet-Variational Counter Net-a model architecture that combines a predictor and a counterfactual generator that are jointly trained, for regression or classification tasks. VCNet is able to both generate predictions, and to generate counterfactual explanations without having to solve another minimisation problem. Our contribution is the generation of counterfactuals that are close to the distribution of the predicted class. This is done by learning a variational autoencoder conditionally to the output of the predictor in a join-training fashion. We present an empirical evaluation on tabular datasets and across several interpretability metrics. The results are competitive with the state-of-the-art method.

Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the limitations of relying solely on their parameters to encode a wealth of world knowledge. This paper aims to understand LMs' strengths and limitations in memorizing factual knowledge, by conducting large-scale knowledge probing experiments of 10 models and 4 augmentation methods on PopQA, our new open-domain QA dataset with 14k questions. We find that LMs struggle with less popular factual knowledge, and that scaling fails to appreciably improve memorization of factual knowledge in the tail. We then show that retrieval-augmented LMs largely outperform orders of magnitude larger LMs, while unassisted LMs remain competitive in questions about high-popularity entities. Based on those findings, we devise a simple, yet effective, method for powerful and efficient retrieval-augmented LMs, which retrieves non-parametric memories only when necessary. Experimental results show that this significantly improves models' performance while reducing the inference costs.

This paper describes Waymo's Collision Avoidance Testing (CAT) methodology: a scenario-based testing method that evaluates the safety of the Waymo Driver Automated Driving Systems' (ADS) intended functionality in conflict situations initiated by other road users that require urgent evasive maneuvers. Because SAE Level 4 ADS are responsible for the dynamic driving task (DDT), when engaged, without immediate human intervention, evaluating a Level 4 ADS using scenario-based testing is difficult due to the potentially infinite number of operational scenarios in which hazardous situations may unfold. To that end, in this paper we first describe the safety test objectives for the CAT methodology, including the collision and serious injury metrics and the reference behavior model representing a non-impaired eyes on conflict human driver used to form an acceptance criterion. Afterward, we introduce the process for identifying potentially hazardous situations from a combination of human data, ADS testing data, and expert knowledge about the product design and associated Operational Design Domain (ODD). The test allocation and execution strategy is presented next, which exclusively utilize simulations constructed from sensor data collected on a test track, real-world driving, or from simulated sensor data. The paper concludes with the presentation of results from applying CAT to the fully autonomous ride-hailing service that Waymo operates in San Francisco, California and Phoenix, Arizona. The iterative nature of scenario identification, combined with over ten years of experience of on-road testing, results in a scenario database that converges to a representative set of responder role scenarios for a given ODD. Using Waymo's virtual test platform, which is calibrated to data collected as part of many years of ADS development, the CAT methodology provides a robust and scalable safety evaluation.